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FOREWORD 


One  problem  faced  by  the  military  is  determining  how  much 
simulation  is  necessary  and  sufficient  for  training  objectives. 
One  issue  in  this  complex  problem  is  that  the  capabilities  for 
simulating  reality  are  increasing  on  an  annual  basis.  Another 
factor  is  that  the  effectiveness  of  the  training  program  is 
directly  related  to  the  instructional  quality  of  the  simulator. 

A  third  issue  is  that  techniques  for  behavioral  analysis  that 
identify  required  features  for  training  devices  exist  but  are 
infrequently  used.  In  addition,  information  on  the  cost- 
effective  use  of  training  devices  within  courses  of  instruction 
is  sparse.  The  development  of  models,  databases,  and  techniques 
addressing  these  issues  will  support  the  design,  fielding,  and 
use  of  advanced  training  technology.  The  potential  effect  on  the 
U.S.  Army  will  be  to  reduce  the  cost  of  fielding  training  devices 
while  increasing  their  instructional  effectiveness. 

The  U.S.  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences  (ARI)  and  the  Simulation,  Training,  and  Instru¬ 
mentation  Command  (STRICOM)  joined  efforts  (Memorandum  of  Agree¬ 
ment  on  Advanced  Technology  for  the  Design  of  Training  Devices, 
1991)  to  investigate  and  develop  models,  databases,  and  analyti¬ 
cal  techniques  that  could  support  the  design  of  advanced  training 
technology. 

STRICOM  has  maintained  partnership  in  the  development  and 
evaluation  of  this  concept  formulation  process  aid  prototype. 

The  concept  formulation  process  aid  (CFP-Aid)  provides  a  basis 
for  supporting  the  integration  of  behavioral  and  engineering 
data,  knowledge,  and  expertise  in  training  device  design.  Final 
product  and  user  evaluation  results  briefings  were  held  in 
December  1991  and  July  1992,  respectively.  Managers  from 
STRICOM' s  Research  and  Engineering  Management  Division  partici¬ 
pated  in  both  briefings.  STRICOM  management  is  currently  consid¬ 
ering  directions  for  application  and  further  development. 
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DEVELOPMENT  OF  A  CONCEPT  FORMULATION  PROCESS  AID  FOR  ANALYZING 
TRAINING  REQUIREMENTS  AND  DEVELOPING  TRAINING  DEVICES 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  perennial  problem  in  training  device  and  simulator 
design  is  meeting  requirements  with  adequate  effectiveness  while 
limiting  cost.  The  Optimization  of  Simulation-Based  Training 
Systems  (OSBATS)  provided  a  theoretically  based  generic  design- 
aid  approach  to  this  problem.  The  next  step  requires  user- 
oriented  implementation  that  aids  decisions  in  the  doctrinally 
mandated  early  phases  of  training  device  design.  The  tradeoff 
determination  (TOD)  phase  of  the  concept  formulation  process 
(CFP)  provides  the  focus  for  developing  a  usable  design  aid.  The 
primary  users  are  engineers  at  the  Simulation,  Training,  and 
Instrumentation  Command  (STRICOM)  who  perform  TOD. 


Procedure : 

A  concept  formulation  process  aid  (CFP-Aid)  prototype  was 
developed  using  the  GURU  development  system  (Micro  Data  Base 
Systems,  Inc.,  1991).  The  GURU  system  incorporates  spreadsheets, 
databases,  graphics,  expert  systems,  text  processing,  and  com¬ 
munications,  along  with  a  fourth-generation  programming  language. 
The  core  material  was  adapted  from  the  OSBATS  program.  The 
OSBATS  models  selected  for  incorporation  into  the  aid  were  based 
on  the  results  of  interviews  with  potential  users  of  the  aid. 

The  CFP-Aid  adopted  a  database  organization  that  incorporates 
data  entry  and  editing  capabilities.  The  aid  supports  the 
development  of  important  user-defined  relational  links  between 
tasks,  cues,  instructional  features,  fidelity  features,  com¬ 
ponents  ,  and  systems . 


Findings: 

The  formative  evaluation  of  the  CFP-Aid  shows  that  the 
system  can  help  a  user  select  training  requirements;  examine 
important  characteristics  of  training  requirements;  identify 
effective  instructional  features  and  fidelity  levels;  perform 
cost,  risk,  and  schedule  analysis;  and  document  both  the  require 
ments  and  analysis  results. 


Utilization  of  Findings: 

The  CFP-Aid  prototype  can  be  used  by  engineers  to  assist  in 
the  TOD  process.  In  addition,  the  system  can  accumulate  and 
maintain  training,  evaluation,  and  training  requirements  informa¬ 
tion.  Initial  use  of  the  system  will  require  additional  effort 
and  guidance,  as  the  database  and  rule  base  structures  are  filled 
and  address  new  application  areas.  Later  use  should  become  much 
more  efficient,  as  usable  task,  component,  and  relational  infor¬ 
mation  is  accumulated. 
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DEVELOPMENT  OF  A  CONCEPT  FORMULATION  PROCESS  AID  FOR  ANALYZING 
TRAINING  REQUIREMENTS  AND  DEVELOPING  TRAINING  DEVICES 

Introduction 

Training  devices  are  designed  and  developed  in  an  iterative 
process  in  which  general  design  concepts  are  successively  refined 
to  produce  detailed  device  descriptions.  At  each  stage  in  this 
process,  concerns  for  cost,  effectiveness,  technological  risk, 
and  development  schedule  are  paramount.  To  satisfy  these 
concerns,  the  evaluation  of  device  concepts  must  determine 
whether  the  design  is  affordable,  whether  it  can  satisfy  the 
training  requirements,  and  whether  it  can  be  produced  on  schedule 
using  existing  or  easily  developed  technology. 

In  previous  work,  the  Human  Resources  Research  Organization 
(HumRRO)  developed  the  Optimization  of  Simulation-Based  Training 
Systems  (OSBATS)  model  as  prototype  software  (Singer  &  Sticha, 
1987;  Sticha,  1989,  1990).  The  OSBATS  program  has  several 
interactive  models  that  evaluate  training  requirements,  specify 
training  device  design  options,  and  prescribe  cost-effective 
designs  in  the  form  of  major  component  descriptions.  However, 
OSBATS  was  designed  from  a  theoretical  viewpoint,  without  any 
specific  user  in  mind  (Singer  &  Sticha,  1992) .  Consequently,  it 
addressed  the  needs  of  specific  participants  in  the  training 
device  design  process  only  in  general  ways. 

This  report  provides  a  summary  of  a  project  that  revised 
portions  of  the  OSBATS  model.  The  goal  was  to  provide  a  useful 
tool  for  engineers  during  the  Trade-off  Determination  (TOD)  phase 
of  the  training  device  Concept  Formulation  process  (CFP) .  The 
project  used  concepts  and  models  from  OSBATS  combined  with  new 
concepts  in  developing  a  system  that  produces  results  that  can 
be  directly  incorporated  into  the  TOD  report. 

Background 

The  OSBATS  prototype  contains  five  modules  that  address 
general  training  device  design  issues: 

1.  Simulation  Configuration  Module.  Task  information  is  used 
to  assign  tasks  to  one  of  three  training  device  approaches: 
part  mission  training  devices,  full-scale  simulators,  or 
actual  equipment.  The  assignment  is  based  on  a  partial 
fidelity  analysis  and  rudimentary  estimates  of  time  and  cost 
savings . 

2.  Instructional  Feature  Selection  Module.  Analyzes  task 
information  to  identify  applicable  instructional  features, 
identifies  the  features,  and  specifies  the  order  (based  on  a 
cost/benefit  analysis,  see  discussion  that  follows)  for 
selection  of  the  instructional  features. 

3.  Fidelity  Optimization  Module.  This  analyzes  task 
information  in  order  to  identify  the  appropriate  fidelity 
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dimensions  and  levels  equired  for  efficient  learning.  The 
routines  then  specify  che  order  (based  on  a  cost/benefit 
analysis,  see  discussion  that  follows)  for  selection  of 
advanced  levels  of  these  dimensions. 

4.  Training  Device  Selection  Module.  This  aids  the  user  in 
determining  the  most  efficient  family  of  training  devices 
(eliminating  redundancies  and  less  efficient  devices) . 

5.  Resource  Allocation  Module.  This  aids  in  determining  the 
optimal  allocation  of  training  time  to  devices  and 
calculates  the  number  of  different  training  devices  required 
to  support  the  training  requirements. 

The  concept  for  operation  of  OSBATS  is  based  on  the  iterative  use 
of  the  five  modules  to  make  recommendations.  Both  the  modules 
that  are  used  and  the  order  in  which  they  are  used  will  vary 
depending  on  the  requirements  of  the  problem  and  the  preferences 
of  the  user. 

The  OSBATS  model  integrates  several  normative  and 
descriptive  modeling  constructs  (Sticha,  1989) .  The  normative 
models  provide  the  structure  for  the  training  system  design 
problem,  specify  a  decision  process,  and  specify  the  requirements 
for  data  content  and  format.  The  descriptive  models  predict 
trainee  performance  and  provide  the  input  to  the  normative 
models.  They  also  define  methods  for  aggregating  available  data 
in  order  to  obtain  values  for  the  parameters  of  the  normative 
model.  The  descriptive  models  provide  a  simple  description  of 
the  complex  processes  that  occur  during  training. 

Normative  modeling.  The  overall  modeling  framework  is  based 
on  methods  that  attempt  to  define  the  training  strategy  that 
meets  the  training  requirements  at  minimum  cost.  This  framework 
was  originally  described  by  Roscoe  (1971) ,  and  has  been  extended 
by  Povenmire  and  Roscoe  (1973),  Carter  and  Trollip  (1980), 

Bickley  (1980),  and  Cronholm  (1985).  It  was  extended  further  to 
provide  the  normative  basis  for  the  OSBATS  model  (Sticha, 
Blacksten,  Buede,  Singer,  Gilligan,  Mumaw,  &  Morrison,  1990) .  In 
its  simplest  form,  the  method  compares  the  ratios  of 
effectiveness  and  cost  for  the  two  training  alternatives.  The 
approach  thus  incorporates  the  trade-off  between  differences  in 
training  benefit  derived  from  the  use  of  simulator  alternatives 
and  differences  in  the  costs  of  using  those  simulators. 

This  simple  formulation  of  training  cost  effectiveness  is 
used  by  the  OSBATS  model  to  generate  an  "optimal"  mix  of 
simulator  and  actual  equipment  training.  In  this  formulation, 
effectiveness  means  the  benefit  or  gain  made  through  using  the 
simulator  or  even  a  different  feature  (e.g.,  an  instructional 
feature  or  a  greater  level  of  fidelity  in  some  component 
dimension) .  This  gain  can  be  in  decreased  time  to  train  to  some 
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standard  or  in  being  able  to  improve  the  performance  standard 
during  the  same  training  time  or  number  of  training  trials.  The 
costs  used  in  the  formulation  are  as  all-encompassing  as 
possible,  including  development,  fielding,  life-cycle,  and 
operational  cost  differentials  (the  difference  between  devices  is 
used  in  the  formulation) .  Another  included  factor  is  that  the 
marginal  change  in  calculated  cost  effectiveness  (improvement 
over  actual  equipment  «-r  another  simulator)  of  new  simulator 
training  is  a  generally  a  decreasing  function  of  the  amount  of 
training  on  that  simulator.  That  is,  the  first  hours  of  training 
using  the  new  simulator  replace  more  training  time  on  actual 
equipment  than  subsequent  hours  do.  This  means  that  at  some 
point  it  becomes  more  cost  effective  to  revert  to  the  actual 
equipment  or  another  simulator.  The  optimal  mix  of  simulator  and 
actual  equipment  training  prescribes  that  new  simulator  training 
be  conducted  until  the  marginal  cost  savings  from  reduced  use  of 
actual  equipment  training  equals  the  marginal  cost  of  using  the 
new  simulator  for  training,  adjusted  for  effectiveness. 

Therefore  the  amount  of  simulator  training  in  the  optimal  mix  is 
a  function  of  the  characteristics  of  the  tasks,  the  capabilities 
of  the  simulator  or  training  device  and  the  actual  equipment,  and 
the  costs  of  using  each  in  training. 

A  second  normative  modeling  aspect,  which  is  based  on 
resource  allocation  methods,  is  used  to  determine  which  device 
capabilities  can  best  meet  the  task  training  requirements  within 
budgetary  constraints.  Two  applications  of  this  method  consider 
instructional  features  and  fidelity  features,  respectively.  Each 
of  these  analyses  considers  independent  features  that  may  be 
either  present  or  absent  in  a  training  device.  The  benefit  of  a 
feature  is  a  mathematical  function  of  the  number  of  tasks  for 
which  the  presence  of  the  feature  can  enhance  training.  The 
ability  of  a  feature  to  enhance  training  for  a  particular  task  is 
derived  from  task  characteristics  using  an  expert  system  rule 
base.  The  analysis  proceeds  by  comparing  the  benefit  of  using 
the  feature  to  the  cost  of  incorporating  the  feature  into  a 
training  device.  The  analysis  then  orders  the  features  by  the 
ratio  of  benefit  to  cost.  This  ordering  specifies  the  optimal 
order  for  selection  of  features  based  on  budget  limits. 

Descriptive  Modeling.  The  descriptive  models  in  the  OSBATS 
model  provide  a  simple  description  of  complex  processes  involved 
in  skill  acquisition  and  transfer.  The  output  of  these  models 
provides  the  critical  information  that  is  used  by  the  normative 
models.  These  models,  in  turn,  provide  logical  methods  for 
aggregating  more  basic  analytic  and  empirical  data,  and  thus 
affect  the  data  requirements. 

The  OSBATS  system  contains  models  that  describe  human 
performance  variables  and  provide  training  cost  estimates.  The 
human  performance  models  characterize  the  acquisition  and 
transfer  processes,  predict  transfer  of  training  as  a  function  of 
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device  and  task  characteristics,  and  predict  training  erficiency 
as  a  function  of  device  instructional  features.  The  predictions 
are  made  based  on  an  acquisition  and  transfer  function  that 
describes  performance  on  actual  equipment  as  a  function  of 
training  time  on  a  training  device,  which  may  also  be  actual 
equipment.  The  acquisition  and  transfer  processes  are 
represented  or  described  by  a  power  function.  The  power  function 
is  characterized  by  an  initially  high  learning  rate  that 
decreases  with  increased  training,  consistent  with  a  long  line  of 
research  that  has  been  summarized  by  Newell  and  Rosenbloom 
(1981)  . 

Problems  With  OS BATS 

The  OSBATS  model  was  innovative  and  revolutionary.  The  goal 
of  the  OSBATS  project  was  to  develop  model  training  device 
specification  procedures  that  were  not  constrained  by  current 
operating  procedures,  organizational  responsibilities,  or  data 
availability.  As  a  result  of  this  goal,  the  procedures 
incorporated  into  the  OSBATS  software  were  not  consistent  with 
existing  methods  and  there  were  considerable  barriers  to  the 
direct  adoption  of  OSBATS  in  the  training  device  design  process. 

A  few  examples  will  highlight  some  of  the  differences 
between  the  procedures  specified  by  OSBATS  and  existing  CFP 
standard  operating  procedures.  First,  the  OSBATS  model  combines 
activities  (such  as  parts  of  simulation  configuration)  that  are 
currently  performed  by  the  schools  before  a  training  device 
requirement  has  been  specified,  activities  (such  as  fidelity 
optimization)  currently  performed  by  STRICOM  as  a  part  of  the 
Trade-Off  Determination  or  Best  Technical  Approach  phase,  and 
activities  (such  as  resource  allocation)  currently  performed  by 
the  schools  after  the  training  device  design  has  been  completed. 
Similarly,  the  data  required  by  the  model  incorporate  knowledge 
of  both  training  specialists  and  engineers,  who  will  generally 
work  for  different  organizations.  Thus,  there  is  no  single  user 
or  user  organization  for  whom  the  OSBATS  model  is  uniquely 
suited. 

Second,  although  OSBATS  can  provide  useful  guidance  for 
various  stages  of  the  training  device  design  process,  it  does  not 
provide  a  complete  product  tor  any  specific  process  or  phase. 

The  incompleteness  of  the  OSBATS  model  is  evident  when  the  model 
capabilities  are  compared  to  the  requirements  of  the  TOD.  The 
OSBATS  model  is  concerned  solely  with  the  ability  of  a  training 
device  to  meet  training  requirements  (effectiveness)  and  the 
lifecycle  cost  of  the  device.  However,  two  additional  concerns 
that  are  critical  to  the  TOD  are  the  technological  risk  involved 
in  the  device  design  and  the  schedule  required  for  development 
and  production  of  the  device.  OSBATS  is  silent  on  these  issues, 
even  though  there  are  significant  interactions  between  risk, 
schedule,  effectiveness,  and  development  costs. 
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The  barriers  to  the  adoption  of  the  OSBATS  methodology  can 
be  summarized  simply  by  stating  that  the  model  was  not  designed 
to  meet  the  needs  of  any  specific  user.  Nevertheless,  many  of 
the  modeling  components  of  OSBATS  are  directly  applicable  to 
problems  of  training  device  design,  and  to  the  TOD  process  in 
particular.  The  goal  of  this  project  is  to  take  appropriate 
elements  of  the  OSBATS  model,  and  combine  them  with  additional 
analyses  to  produce  a  decision  aid  that  is  tailored  to  the 
requirements,  procedures,  and  products  of  the  TOD  process. 

Organization  of  This  Report 

The  remainder  of  this  report  describes  the  design, 
development,  and  formative  evaluation  of  a  Concept  Formulation 
Process  Aid  (CFP-Aid)  for  the  TOD  process.  The  next  section  of 
the  report  describes  the  identification  of  processes  that  could 
be  aided,  and  the  plan  for  developing  the  aid.  The  design  of  the 
aid  was  based  on  interviews  with  STRICOM  engineers,  review  of  TOD 
documentation,  and  knowledge  of  OSBATS  trade-offs.  CFP-Aid 
incorporates  rewritten  software  algorithms  and  heuristics  from 
OSBATS.  It  also  adds  new  analyses  and  provides  a  flexible  basis 
for  data  organization.  Formative  evaluation  of  the  system  during 
development  was  based  primarily  on  demonstrations  and  interviews 
with  prospective  users.  The  software  is  described  in  the  third 
section  of  the  report.  That  section  covers  the  goals  and  major 
elements  of  the  CFP-Aid.  The  next  section  presents  the  results 
of  detailed,  summative  evaluations.  The  final  section  of  the 
report  provides  suggestions  for  further  research  and  final 
conclusions  from  the  development  effort. 

Designing  the  CFP-Aid 

Design  phase  activities  focused  on  identifying  the 
supporting  processes  to  be  included  in  the  CFP-Aid  and 
determining  how  to  implement  the  prototype  software.  The  design 
phase  involved  the  following  activities. 

•  identifying  the  specific  activities  that  could  be  supported 
by  a  CFP-Aid  for  Trade-Off  Determination  (TOD) , 

•  determining  the  tools  in  the  OSBATS  model  that  could  help 
the  engineer  perform  identified  TOD  activities, 

•  proposing  new  tools  and  analyses  that  could  facilitate  the 
TOD  process ,  and 

•  developing  and  organizing  proposed  CFP-AID  support 
processes . 

Prototype  implementation  was  carried  out  using  the  GURU* 
development  system  (Micro  Data  Base  Systems,  Inc.,  1991).  GURU 
is  an  integrated  system  that  includes  integrated  database, 
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spreadsheet,  text  processing,  expert  system  shell,  graphics,  and 
communication  systems.  The  development  system  also  includes  an 
interpreted  fourth  generation  language,  which  makes  it  a  good 
environment  for  prototype  development.  The  government  had 
previously  tested  the  transition  of  portions  of  OSBATS  to  the 
development  system.  Those  activities  provided  a  basis  for 
familiarizing  the  development  team  with  the  capabilities  of  the 
GURU  system  that  could  be  exploited  in  the  CFP-Aid  prototype. 

Identification  of  TOD  Processes 


The  TOD  activities  selected  for  the  CFP-Aid  were  identified 
through  interviews  with  potential  users  of  the  aid,  review  of  CFP 
documentation,  and  analyses  of  existing  and  possible  procedures 
to  support  the  CFP.  The  result  of  this  analysis  was  a  list  of 
proposed  supporting  processes  for  the  CFP-Aid.  The  proposed 
supporting  processes  were  then  reviewed  by  users,  who  prioritized 
them.  The  following  subsections  describe  design  phase  activities 
in  greater  detail. 

User  interviews  and  document  review.  Review  of  STRICOM  user 
interview  transcripts  provided  an  important  source  of  information 
about  TOD  processes.  These  interviews  addressed  how  the  Concept 
Formulation  Process  is  currently  conducted,  what  data  are 
available,  and  what  types  of  assistance  users  would  like  to 
receive.  The  review  indicated  that  there  is  tremendous  variety 
in  the  procedures  that  are  used  during  the  TOD  phase.  The 
general  opinion  among  interviewees  was  that  there  are  few 
generalities  in  the  TOD  process.  The  process  is  viewed  as 
depending  on  the  specific  nature  of  the  training  device  need,  the 
relevant  school,  and  the  engineer  performing  the  analysis.  Our 
examination  of  the  provided  TOD  documentation  examples  confirmed 
that  there  was  considerable  variation  in  the  scope  and  depth  of 
the  analysis,  the  number  of  device  configuration  options  and 
evaluation  factors  considered,  and  the  formality  of  the  analysis 
procedures . 

However,  some  common  methods  and  considerations  were 
identified  in  the  review  of  user  interviews  and  example 
documentation.  The  first  critical  concern  in  the  TOD  process  is 
to  identify  the  major  component  categories  required  (e.g.,  visual 
systems,  motion  systems,  equipment  mock-up)  and  the  possible 
levels  of  simulation  in  each  of  these  categories  (referred  to  as 
families  in  the  CFP-Aid) .  The  second  step  is  to  estimate, 
assign,  and  develop  a  trade-off  for  the  cost,  technical  risk,  and 
development  schedule  for  the  individual  configuration  options 
(e.g.,  different  visual  system  field  of  view  options).  The 
engineers  use  their  expertise  to  identify  possible  options  that 
would  effectively  apply  to  the  training  requirements.  Once  these 
options  are  identified,  the  trade-off  determination  is 
structured.  (The  Trade-Off  Analysis  or  TOA  is  the  phase 
immediately  following  the  TOD,  and  is  performed  by  the  training 
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device  proponent  based  on  the  TOD  information  and  structure.) 
Often,  in  structuring  the  TOD,  both  technical  risk  and 
development  schedule  are  simply  assessed  on  rating  scales  with 
five  or  fewer  categories  (e.g.,  low,  medium-low,  medium,  medium- 
high,  high)  by  the  engineer.  The  cost  estimates  typically 
address  the  research  and  development,  procurement,  construction, 
installation,  and  sustainment  costs  required  for  each  option.  In 
general,  the  TOD  process  at  STRICOM  does  not  question  the 
capabilities  specified  in  the  training  device  requirement  (from 
the  school  or  the  major  weapon  systems  project  officer)  unless 
those  capabilities  are  demonstrably  infeasible  in  terms  of  cost, 
schedule,  or  technical  risk.  Consequently,  many  of  the  fidelity 
considerations  addressed  by  STRICOM  (e.g.,  resolution  in  visual 
systems)  only  compare  fidelity  levels  greater  than  those 
specified  in  the  requirements. 

Review  of  OSBATS  capabilities.  We  reviewed  the  capabilities 
of  the  OSBATS  model  to  identify  those  portions  that  would  be  most 
useful  in  supporting  the  TOD  processes.  Based  on  this  analysis, 
and  on  reviews  of  the  interview  transcripts,  we  developed  a  list 
of  candidate  operations  for  the  CFP-Aid.  The  major  concern  of 
TOD  is  the  identification  and  evaluation  of  major  design  options, 
as  described  above.  Consequently,  the  OSBATS  Training  Device 
Selection  and  Resource  Allocation  modules  were  not  judged  as 
useful  for  adaptation  into  a  TOD  decision  aid.  Those  modules  are 
more  concerned  with  the  analysis  of  a  set  of  training  devices  in 
order  to  determine  the  best  set,  order  of  use,  and  device  use 
time  that  supports  the  efficient  acquisition  of  a  group  of  tasks. 

The  device  design  modules  from  OSBATS  (Instructional 
Features  Selection  and  Fidelity  Optimization)  were  appropriate 
for  adaptation,  although  they  use  greater  detail  than  is  required 
for  this  stage.  In  addition,  these  rule  bases  are  specific  to  a 
category  of  task  types  (i.e.,  helicopter  pilot  training),  and 
need  to  be  supplemented  with  other  ways  to  obtain  requirements 
for  training  device  components  (major  pieces  of  the  training 
device) . 

The  equipment  checklist  in  the  OSBATS  Simulation 
Configuration  module  represents  activities  that  are  generally 
conducted  before  the  TOD.  However,  because  it  may  provide  some 
utility  as  a  memory  aid  and  was  judged  to  be  very  easy  to 
implement  in  the  current  aid,  it  was  included. 

Selecting  CFP-Aid  operations.  Candidate  operations  for  the 
CFP-Aid  were  listed  and  organized  at  a  meeting  attended  by  both 
HumRRO  and  ARI  researchers.  The  candidates  were  integrated  into 
a  single  list,  and  preliminary  assessments  of  cost  (of 
development  or  translation  from  OSBATS)  and  benefit  (to  the  TOD 
process)  were  made.  Operations  that  represented  straightforward 
translations  of  OSBATS  capabilities  were  judged  to  have  low  cost. 
Any  development  of  new  capabilities  was  judged  to  have  high  cost. 
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The  benefits  were  tentatively  assigned  at  this  point,  and  were 
subject  to  STRICOM  review. 

STRICOM  engineers  reviewed  the  integrated  list  and  sorted 
the  proposed  CFP-Aid  support  operations  according  to  priority. 
Priority  was  rated  as  high,  medium,  or  low  for  each  of  the 
candidates.  The  STRICOM  input  was  used  to  adjust  the  benefits 
and  to  select  the  operations  for  inclusion  in  the  final  design. 

CFP-Aid  Development  Plan 

During  the  design  phase,  we  reviewed  the  capabilities  of  the 
GURU  development  system  (Micro  Data  Base  Systems,  Inc.,  1991)  so 
that  we  could  exploit  the  strengths  of  the  development  system  in 
the  CFP-Aid  design.  As  part  of  this  review,  we  examined 
demonstration  software  that  ARI  developed  to  illustrate 
capabilities  of  the  GURU  system  and  possible  interface  options 
for  the  CFP-Aid.  The  demonstration  software  focused  on  system 
organization  and  implemented  portions  of  the  Instructional 
Feature  and  Fidelity  Optimization  modules.  It  illustrated  the 
capabilities  of  the  GURU  development  environment,  and  showed  how 
OSBATS  modules  could  be  translated  into  the  GURU  environment. 

The  product  of  the  design  phase  was  a  development  plan.  The 
plan  described  both  the  supporting  operations  to  be  included  in 
the  initial  CFP-Aid  prototype  and  an  identification  of  operations 
that  could  be  incorporated  as  later  enhancements.  In  fact,  many 
of  these  enhancements  are  included  in  the  prototype  CFP-Aid.  The 
plan  also  described  the  top  level  of  the  user  interface.  The 
user  interface  is  designed  around  a  set  of  pull-down  menus  that 
represent  the  general  classes  of  operations.  The  complete 
details  of  the  menu  structure  are  presented  in  the  CFP-Aid  User's 
Guide  (Elder,  Sticha,  Page,  and  Singer,  1993) . 

The  development  plan  enumerated  both  file  management  and 
analytical  operations  that  were  planned  for  the  CFP-Aid.  The 
file  management  operations  are  required  to  start  new  analyses, 
open  files,  save  and  delete  files,  and  print  information.  The 
planned  analytical  operations  would  perform  the  following 
analyses  to  support  the  TOD  process. 

1.  Identify  Tasks.  This  module  allows  the  user  to  select  tasks 
from  a  master  database  or  to  enter  them  and  their  associated 
data  directly.  It  provides  information  about  the  tasks 
associated  with  the  Military  Occupational  Specialty  (MOS) 
and  associated  Additional  Skill  Indexes  (ASI)  and  Skill 
Qualification  Indexes  (SQI)  for  which  training  must  be 
provided.  From  that  information,  the  user  may  select  the 
tasks  and  associated  information  that  will  be  used  to 
identify  the  required  training  device  components. 
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2. 


Determine  cue  sources.  This  module  allows  the  user  to 
select  or  enter  the  types  of  cues  (visual,  auditory, 
proprioceptive,  kinesthetic,  etc.)  and  the  sources  of 
information  (terrain,  instruments,  cockpit,  system  motion, 
etc.)  that  are  required  to  perform  the  activities  involved 
in  a  task.  Baseline  information  will  provide  a  preliminary 
set  of  cue  sources,  which  the  user  may  edit. 

3.  Simulation  checklist.  This  module  allows  the  user  to 
identify  those  tasks  that  are  the  specific  targets  of 
simulated  training  because  of  safety  concerns,  need  for 
special  training  conditions,  or  the  possibility  of 
improvements  in  training  efficiency.  This  essentially 
replicates  the  OSBATS  Simulation  Configuration  module,  which 
was  used  to  ensure  that  tasks  required  simulation. 

4.  Instructional  feature  requirements.  This  module  applies  a 
set  of  rules  that  identify  the  instructional  support 
features  required  for  a  training  device.  The  rules  require 
information  about  the  characteristics  of  the  tasks  being 
trained.  This  capability  was  available  as  the  instructional 
feature  rule  base  in  the  OSBATS  model. 

5.  Fidelity  features  requirements.  This  module  applies  a  set 
of  rules  that  identify  the  fidelity  features  required  for  a 
training  device.  The  rules  require  information  about  the 
characteristics  of  the  tasks  to  be  trained.  This  capability 
was  available  as  the  fidelity  rule  base  in  the  OSBATS  model. 

6.  Select  device  components.  This  module  allows  the  user  to 
select  training  system  components  that  will  compose 
candidate  system  designs  to  be  evaluated  in  the  TOD.  The 
components  may  be  based  on  the  previous  rule  bases,  the  user 
may  select  the  components  directly,  or  may  define  new 
components . 

7.  Component  cost,  effectiveness,  risk,  and  schedule  (CERS) 
analysis.  This  module  provides  an  analysis  that  supports  a 
component  by  component  CERS  trade-off.  The  analysis 
operations  are  based  on  engineer  models  and  recommendations. 

8.  Define  system  alternatives.  This  module  assists  the  user  in 
combining  components  to  develop  system  alternatives. 

9.  System  CERS  analysis.  This  module  supports  CERS  analysis  at 
the  system  level.  It  assists  the  user  in  assessing  and 
comparing  cost,  risk,  and  schedule  between  whole  system 
alternatives  developed  in  the  Define  system  alternatives 
module . 

Three  of  the  nine  modules  included  in  the  development  plan 

were  obtained  directly  from  OSBATS  capabilities.  Those  are  the 
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simulation  checklist,  the  instructional  feature  rule  base,  and 
the  fidelity  rule  base.  The  other  supporting  operations  were 
identified  from  user  interviews  and  review  of  TOD  documentation. 

Since  we  employed  a  prototyping  approach  to  the  development 
of  the  CFP-Aid,  the  initial  plan  could  not  completely  specify  the 
design  of  the  CFP-Aid  system.  During  the  development  phase,  the 
design  was  modified  several  times  as  new  operations  were 
identified  or  redundant  operations  eliminated.  Many  of  the 
changes  were  minor,  and  covered  such  topics  as  the  naming  of  menu 
items,  project  subdirectories,  and  so  forth.  Other  changes  were 
of  greater  scope.  For  example  the  system  was  altered  so  that 
instructional  feature  and  fidelity  rule  bases  could  take  input 
data  from  either  tasks  or  cues.  In  addition,  fidelity  rule  bases 
may  take  data  from  instructional  features. 

Software  Description 

The  CFP-Aid  software  is  used  in  the  TOD  phase  of  the  CFP  in 
the  training  device  development  process.  This  software  aid 
assists  the  designer  in  proceeding  systematically  through  the 
stages  of  trade-off  determination. 

CFP-Aid  Goal 

The  goal  of  the  CFP-Aid  is  to  assist  the  engineer  perform 
the  activities  of  the  TOD.  The  following  activities  are 
specifically  addressed  by  the  CFP-Aid. 

•  Selection  of  training  requirements. 

•  Examination  of  training  requirements. 

•  Identification  of  effective  instructional  features  and 
fidelity  levels. 

•  Performance  of  cost,  risk,  and  schedule  analysis. 

•  Documentation  of  requirements  and  analysis  results. 

Each  CFP-Aid  capability  is  designed  to  support  some  TOD  activity. 
The  basic  operation  of  the  CFP-Aid  is  organized  around  data 
elements  which  are  connected  by  links.  The  following  discussion 
describes  CFP-Aid  operations  in  this  context. 

Elements  9f  the  CFP-Aid 

To  understand  CFP-Aid  operations,  it  is  first  necessary  to 
become  familiar  with  several  terms  that  describe  elements  of  the 
analyses  performed  by  the  aid.  The  terms  and  the  relationships 
among  them  are  shown  in  Figure  1;  each  of  them  is  briefly  defined 
below. 
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Figure  1.  CFP-Aid  data  elements  and  links. 


Tasks  and  Functions.  A  task  is  the  smallest  activity  that 
performs  a  meaningful  job  function  (Department  of  Defense,  1986) . 
Two  examples  of  tasks  are  "perform  terrain  flight  approach"  and 
"clear  the  M-240  machinegun."  In  the  CFP-Aid,  the  definition  of 
function  is  drawn  from  conversation  with  engineers.  Function 
(when  used  in  this  document  in  reference  to  the  CFP-Aid)  refers 
to  any  common  activity  or  requirement  (e.g.,  for  visual  input) 
for  one  or  more  tasks.  An  example  of  a  function  is  "weapons  and 
emergency  procedures."  This  function  can  be  used  to  address 
several  tasks  because  they  include  procedures  and  have  similar 
training  device  requirements.  For  example,  none  of  these  tasks 
requires  a  sophisticated  visual  display  system  or  complex  force 
feedback  on  the  controls.  On  the  other  hand,  tasks  in  this  group 
all  require  controls  and  displays  that  interact  appropriately 
(functional  fidelity),  and  that  are  positioned  as  they  would  be 
in  the  actual  equipment  (physical  fidelity;  Hays  &  Singer,  1988) . 
Therefore  these  tasks  have  comonalities  and  can  be  treated  as  a 
single  requirement,  referred  to  as  a  function. 

Cues/Requirement  category.  The  requirement  category  is  the 
category  of  information  that  is  required  to  perform  a  task  or 
function.  This  information  provides  critical  cues  or  response 
feedback  necessary  for  effective  learning  of  the  task  or 
function.  For  example,  abnormal  engine  operating  noise  may 
provide  a  cue  for  the  initiation  of  an  emergency  procedure,  and 
should  therefore  be  provided  so  that  the  relationship  can  be 
learned  on  the  training  device.  Similarly,  force  feedback  on 
controls  may  be  required  to  learn  low  altitude  flight  tasks. 
Requirement  categories  are  often  referred  to  only  as  "cues"  both 
in  the  CFP-Aid  and  in  this  report. 
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Instructional  feature.  An  instructional  feature  is  a 
training  device  capability  that  helps  an  instructor  manage 
training  or  organize  training  activities  to  enhance  efficiency. 
Instructional  features  are  generally  unrelated  to  the  realism  of 
a  simulation,  although  in  some  cases  an  instructional  feature 
(such  as  augmented  feedback)  may  enhance  or  reduce  realism  at 
appropriate  stages  of  training  for  instructional  benefit. 
Automated  performance  measurement  (which  measures  performance 
according  to  some  preset  standard)  is  an  example  of  an 
instructional  feature  that  does  not  interact  with  fidelity. 

Fidelity  feature.  A  fidelity  feature  is  a  dimension  on 
which  training  devices  can  present  stimuli  or  response  options  to 
students  with  greater  or  lesser  realism.  Examples  of  fidelity 
issues  are  the  resolution  of  the  visual  display  or  the  number  of 
degrees  of  motion  provided  by  the  platform  motion  system.  Often 
a  certain  degree  of  realism  is  required  for  training  on  a 
training  device  to  transfer  successfully  to  performance  on  actual 
equipment.  The  level  of  fidelity  required  depends  on  the 
specific  aspects  of  the  task  or  functions  being  trained  to 
students  and  the  cues  required  to  learn  to  perform  those  tasks  or 
functions . 

Component .  The  component  is  the  part  of  a  training  system 
that  represents  an  approach  to  performing  a  device  function.  For 
the  purposes  of  the  CFP-Aid,  components  are  the  alternatives  that 
are  addressed  in  the  TOD.  In  performing  the  TOD,  components  that 
address  some  aspect  of  the  requirement  (such  as  alternative 
approaches  to  the  required  visual  display)  are  compared.  The 
CFP-Aid  considers  the  components  that  address  the  same 
requirement  to  be  in  the  same  family.  Examples  of  component 
families  considered  in  the  CFP-Aid  prototype  are  visual  displays, 
image  generation,  sound  generation,  platform  motion,  and  seat 
motion.  Components  within  a  family  may  differ  according  to  how 
well  they  address  one  or  more  fidelity  requirements.  For 
example,  platform  motion  components  differ  in  the  degrees  of 
freedom;  image  generation  systems  differ  in  visual  resolution, 
update  rate,  and  visual  content.  Other  components,  such  as  an 
Instructional  Management  Computer,  may  be  required  to  support  the 
instructional  features  of  the  training  device. 

System.  A  system  is  a  collection  of  components,  one  from 
each  required  family,  constituting  a  complete  training  device 
design  alternative.  Systems  with  different  combinations  of 
components  are  compared  in  a  TOD.  A  combat  mission  trainer  with 
helmet -mounted  display,  a  computer  generated  imagery  system,  a 
six  degrees-of-freedom  motion  system,  and  possibly  other 
components  as  well,  is  an  example  of  a  system.  This  system  would 
be  compared  to  another  system  in  which  different  components  were 
selected  for  each  family.  For  example,  the  comparison  system 
could  include  a  less  capable  computer  generated  imagery  system, 
and/or  a  dome  visual  display  system. 
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CFP-Aid  Data  Element  Links 


The  elements  of  the  CFP-Aid  are  associated  by  links  as  shown 
in  Figure  1.  The  links  represent  the  inferential  network  that 
guides  the  operation  of  the  aid.  For  example,  tasks  are  linked 
to  requirement  categories  (or  cues) ,  instructional  features, 
fidelity  issues,  and  components.  The  existence  of  these  links 
means  that  knowledge  of  the  task  may  have  implications  on  any  of 
the  four  other  elements  to  which  tasks  are  linked.  A  particular 
task  may  require  a  certain  kind  of  cue,  or  have  specific 
requirements  for  visual  resolution,  platform  motion,  or  other 
fidelity  issue.  Similarly,  the  task  may  require  that  a  specific 
instructional  feature  or  component  be  included  in  any  device  that 
is  used  to  train  the  task. 

The  types  of  links  that  have  been  included  in  the  model 
provide  it  with  flexibility  that  is  appropriate  for  both  the 
early  stage  of  the  device  design  process  and  the  limited  research 
knowledge  regarding  device  requirements.  At  this  stage  in  the 
process,  it  may  not  always  be  possible  to  use  a  single 
inferential  chain  to  obtain  all  device  requirements.  For 
example,  there  is  a  rich  body  of  knowledge  regarding  fidelity 
issues  related  to  the  visual  and  motion  components  of  flight 
trainers.  This  knowledge  is  especially  useful  for  situations  in 
which  the  flight  tasks  are  relatively  well  understood  and  similar 
to  tasks  that  have  been  addressed  in  the  research  literature. 
However,  if  at  the  time  the  TOD  is  performed  the  targeted  tasks 
are  only  partially  understood  then  the  requirements  for  device 
components  must  be  determined  using  some  consideration  other  than 
fidelity  issues. 

The  CFP-Aid  contains  three  lines  of  reasoning  that  can  meet 
the  needs  of  the  TOD:  (a)  inferences  may  be  made  directly  from 
task  or  function  requirements;  (b)  inferences  may  be  made  by 
considering  the  cues  to  which  tasks  or  functions  are  linked;  and 
(c)  inferences  may  be  made  considering  instructional  features  to 
which  tasks  or  functions  are  linked.  In  actual  applications  of 
the  CFP-Aid,  we  suspect  that  multiple  lines  of  reasoning  will  be 
employed.  The  eleven  lines  connecting  the  elements  in  Figure  1 
represent  the  inferential  links  in  the  CFP-Aid.  The  following 
discussions  describe  each  of  these  links. 

Function-Cue  links.  A  function  is  linked  to  a  particular 
cue  if  the  cue  is  required  to  perform  that  function.  For 
example,  the  function,  perform  emergency  procedures,  may  be 
linked  to  visual  cues,  auditory  cues,  motion  cues,  and/or  cues 
from  cockpit  instruments.  In  an  actual  problem,  the  cues  would 
be  specified  in  greater  detail  than  in  the  above  example. 

Function-Instructional  feature  data  links.  These  links  are 
between  task/ functions  and  data  records  which  contain  specific 
information  required  by  the  instructional  feature  rule  base. 
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This  allows  the  rule  driven  system  to  identify  instructional 
features  based  on  the  information  about  the  task/ function. 

Function-Fidelity  data  links.  The  function-fidelity  data 
links  are  used  by  fidelity  rule  bases.  There  are  currently  two 
types  of  fidelity  rule  bases,  one  that  addresses  motion  and  one 
that  addresses  visual  issues.  Functions  are  linked  to  data 
records  tailored  for  one  or  more  of  the  fidelity  rule  bases.  For 
example,  a  function  might  be  linked  to  a  particular  data  record 
that  described  the  size  and  distance  of  objects  that  must  be 
detected  to  perform  a  particular  function.  The  visual  rule  base 
would  then  analyze  the  data  record  to  recommend  the  visual 
resolution  that  would  be  required  to  present  simulated  objects  of 
the  required  size  and  distance.  As  fidelity  rule  bases  are 
added,  fidelity  data  records  must  be  formatted  and  the  capability 
for  linking  to  the  appropriate  task  or  function  must  be  provided. 

Function-Component  1 inks .  This  link  allows  a  direct 
inference  from  function  to  components.  This  link  may  represent  a 
command  directive  or  other  user  directed  association  between 
functions  and  components  that  are  not  created  by  the  system 
linking  mechanisms  (e.g.,  rulebases) .  The  presence  of  this  link 
allows  the  user  to  tailor  the  analysis  of  the  CFP-Aid  to  ensure 
that  a  particular  component  is  required  by  a  given  function. 

Cue-Instructional  feature  data  links.  This  link  is 
structured  in  the  same  fashion  as  the  function- instructional 
feature  data  links,  and  can  be  used  in  the  same  way.  The  CFP-Aid 
currently  has  no  instructional  feature  rule  base  that  is  designed 
to  evaluate  the  instructional  feature  requirements  of  individual 
cues.  However,  this  link  provides  the  capability  for  future 
expansion  in  that  area. 

Cue-Fidelity  feature  data  links.  This  link  is  structured  in 
the  same  fashion  as  the  function-fidelity  feature  data  links. 

The  CFP-Aid  currently  has  no  fidelity  rule  bases  that  are 
designed  to  evaluate  the  fidelity  requirements  for  individual 
cues.  As  is  the  case  for  the  cue- instructional  feature  link, 
this  link  provides  the  capability  for  future  expansion. 

Cue-Component  links.  Cues  can  be  linked  directly  to 
components.  For  example,  if  performing  a  function  requires  a 
specific  auditory  cue,  such  as  the  sound  of  a  particular  engine 
malfunction,  then  a  training  device  for  that  function  must 
incorporate  a  component  that  can  produce  that  cue.  In  this  case, 
the  cue  would  be  linked  to  all  components  (in  an  auditory  family) 
that  can  produce  the  required  cue.  This  allows  the  user  to 
initiate  the  function-cue  link,  and  automatically  obtain  all 
required  components  addressing  the  requirements  of  that  cue. 

Instructional  feature-Fidelitv  feature  links.  This  link  is 
available  for  establishment  by  the  fidelity  rule  bases  in  much 
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the  same  fashion  as  other  links  to  fidelity  features. 
Theoretically,  if  inclusion  of  an  instructional  feature  would 
have  an  effect  on  a  fidelity  dimension,  the  information  about 
that  instructional  feature  would  be  used  by  rules  to  evaluate  the 
fidelity  requirements.  The  CFP-Aid  currently  has  no  fidelity 
rule  bases  that  are  designed  to  evaluate  the  fidelity 
requirements  of  individual  instructional  features.  This 
capability  is  provided  for  future  expansion. 

Instructional  feature-component  links.  This  link  addresses 
the  component  implications  of  specific  instructional  features. 
Some  instructional  features  require  hardware  components  such  as 
instructor/operator  stations  or  course  management  computers. 

Still  other  instructional  features  require  software  components 
which  may  or  may  not  be  of  sufficient  magnitude  to  be  a 
consideration  in  the  TOD.  This  link  allows  the  identification  of 
special  component  considerations  based  on  required  instructional 
features. 

Fidelity  feature-component  links.  The  results  of  the 
operation  of  fidelity  rule  bases  are  linked  directly  to  all 
components  that  satisfy  that  particular  fidelity  requirement. 

For  example,  if  the  motion  rule  base  indicates  that  platform 
motion  is  required,  but  that  three  degrees  of  freedom  is 
sufficient,  then  the  results  will  be  linked  to  all  platform 
motion  components  offering  three  degrees  of  freedom  or  greater. 
Thus  the  links  indicate  the  sufficiency  of  the  component,  rather 
than  its  necessity. 

Component-System  links.  Component-system  links  are 
generated  when  CFP-Aid  is  run.  The  aid  can  generate  all  possible 
combinations  of  candidate  components  to  form  system  alternatives. 
The  total  number  of  system  alternatives  is  the  product  of  the 
number  of  available  components  in  each  component  family,  which 
can  be  extensive  if  there  are  several  options  for  each  family. 
Usually,  the  user  will  eliminate  many  of  these  alternatives 
before  conducting  any  analysis.  The  links  define  the  system 
alternatives  in  terms  of  the  components. 

CFP-Aid  Processes 

Select  training  requirements.  The  CFP-Aid  supports  the 
selection  of  training  requirements  by  three  methods:  (a)  direct 
entry  of  required  tasks  or  functions,  (b)  selection  of  training 
requirements  from  a  master  list  of  functions,  and  (c)  selection 
of  training  requirements  based  on  selected  Military  Occupational 
Specialty  (MOS) ,  Additional  Skill  Indicator  (ASI) ,  and  Skill 
Qualification  Indicator  (SQI) .  Selected  training  requirements 
may  be  reviewed  and  modified  at  any  time.  In  addition,  the  user 
may  use  several  MOS,  ASI,  or  SQI  as  the  basis  of  the  training 
requirements  by  performing  multiple  selections. 
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The  task/ functions  database  is  designed  to  be  cumulative. 
That  is,  the  more  the  aid  is  used  and  the  more  tasks  and 
functions  are  added,  the  more  useful  it  should  become.  Continued 
use  will  create  additional  functions  linked  to  additional  MOS, 

ASI ,  and  SQI,  and  more  functions  and  cues  linked  to  device 
components.  This  in  turn  will  make  the  identification  and 
selection  of  training  requirements  easier  and  the  aid  more 
valuable  to  the  developer. 

Examine  training  requirement.  The  system  helps  organize  and 
examine  training  requirements  by  supporting  the  specification  of 
requirement  categories  or  cues.  The  cues  represent  information 
that  is  required  for  an  individual  to  perform  a  specific 
task/ function.  The  CFP-Aid  supports  three  methods  of  selecting 
cues:  (a)  The  CFP-Aid  may  select  cues  automatically,  based  on 

existing  links  from  tasks/functions;  (b)  the  user  may  select  cues 
from  a  master  list  of  cues,  creating  a  new  set  of  links;  and  (c) 
the  user  may  add  new  cues  to  the  database  directly,  linking  these 
cues  to  specific  tasks  or  functions.  The  user  may  review  and 
modify  the  list  of  selected  cues  at  any  time,  and  add  or  delete 
cues  as  necessary. 

Identify  effective  instructional  features  and  fidelity 
levels.  The  aid  will  help  the  user  to  (a)  identify  instructional 
features  through  the  rules  incorporated  into  the  system,  and  (b) 
identify  required  fidelity  levels  for  visual  and  motion 
characteristics  (the  rule  bases  currently  implemented) .  (For 
information  about  adding  further  rule  bases  to  the  system,  see 
Appendix  C,  Page,  Blacksten,  Elder,  Sticha,  &  Singer,  in 
publication) .  The  system  uses  the  rule  bases  to  link 
requirements  to  candidate  system  component  alternatives.  The 
user  may  also  directly  create  links  from  functions,  cues, 
instructional  features,  or  fidelity  features  to  candidate 
components . 

The  expert  system  rule  bases  operate  in  three  modes: 
automatically  based  on  preselected  and  stored  data,  automatically 
with  user  confirmation  of  stored  data,  and  manually  based  on  user 
supplied  data.  In  automatic  operation,  uhe  rule  base  is 
consulted  using  data  from  the  CFP-Aid  database.  No  user 
interaction  is  required,  unless  the  values  for  required  data  are 
not  known.  The  automatic,  with  confirm,  option  is  the  same  as 
automatic  operation,  except  that  the  user  can  examine  the  input 
data  for  each  rule  before  the  rule  is  used.  The  user  may  make 
changes  to  the  input  data  at  that  time.  In  the  third  option,  the 
user  supplies  all  of  the  data  required  by  the  rule  base, 
answering  the  rule  driven  questions  one  by  one. 

The  instructional  features  analysis  is  based  on  the  rule 
base  developed  for  the  OSBATS  model.  Recommendations  are  made 
for  each  task  or  function  based  on  instructional  feature  data. 
Although  the  CFP-Aid  allows  links  from  cues  to  instructional 
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features,  none  of  these  links  are  present  in  the  delivered 
database  because  the  instructional  feature  rule  base  was  designed 
to  consider  task  data.  Making  instructional  feature 
recommendations  regarding  cues  would  require  developing  a  new 
instructional  feature  rule  base,  or  modifying  the  existing  rule 
base,  to  accept  cue  specific  information. 

As  discussed  previously,  the  aid  currently  contains  fidelity 
rule  bases  for  identifying  only  motion  and  visual  components. 
These  two  rule  bases  were  obtained  by  revising  portions  of  the 
fidelity  rule  base  developed  in  the  OSBATS  model.  The  structure 
of  the  CFP-Aid  is  designed  to  support  additional  rule  bases  as 
they  are  developed.  These  rule  bases  make  fidelity 
recommendations  for  each  function  based  on  fidelity  specific 
data.  The  fidelity  rule  bases  also  work  in  the  three  modes 
discussed  above:  automatically,  automatically  with  user 
confirmation,  and  manually  with  user  supplied  data. 

The  aid  also  provides  justification  for  the  recommendations 
made.  The  justification  consists  of  a  brief  description  of  each 
rule  that  was  used  to  select  the  instructional  features  that  were 
required  for  a  function.  Justification  may  be  examined  for  a 
single  function  or  cue,  as  selected  by  the  user. 

Finally,  the  user  may  also  specify  direct  links  between 
functions  and  components.  This  capability  is  implemented  in  a 
general  data  maintenance  procedure  that  allows  the  user  to  enter 
new  function  data  or  edit  the  existing  function  data. 

Assist  in  Cost.  Effectiveness.  Risk,  and  Schedule  ( CERS) 
analysis.  The  system  aids  evaluation  of  component  cost, 
effectiveness,  risk,  and  schedule  factors  by  displaying  graphs 
and  tabular  charts  for  each  factor.  Graphs  and  tabular  displays 
generated  by  the  CFP-Aid  can  be  used  to  build  training  device 
system  alternatives  from  selected  components.  Each  factor  in  the 
CERS  analysis  can  be  examii.ad  separately  in  a  graph  and  in  a 
spreadsheet.  Similar  displays  are  available  at  the  training 
device  system  level.  In  addition,  the  CFP-Aid  supports  trade-off 
analyses  at  that  level.  The  trade-off  analyses  generate  a 
weighted  average  of  cost,  effectiveness,  risk,  and  schedule 
factors.  CFP-Aid  allows  the  user  to  tailor  th  j  analyses  by 
editing  the  estimates  or  the  weights  (importance)  for  these 
factors . 

The  factors  of  the  training  device  system  CERS  are  derived 
from  the  corresponding  factors  in  the  component  analyses.  System 
cost  is  obtained  by  summing  the  cost  of  the  components  included 
in  a  system.  System  effectiveness  is  obtained  by  finding  the 
total  number  of  tasks,  cues,  instructional  features,  and  fidelity 
features  that  link  or  point  (as  described  in  the  linking 
structures,  above)  to  one  of  the  components  in  the  system.  Risk 
is  combined  as  a  probability  of  development  failure;  that  is,  the 
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probability  of  development  failure  for  the  entire  training  device 
is  the  highest  probability  assigned  to  components  development 
failure.  Finally,  schedule  is  combined  by  simply  assuming  that 
all  component  Research  and  Development  (R&D)  activities  are 
conducted  in  parallel;  the  overall  schedule  is  therefore  the 
longest  of  the  component  schedules. 

An  overall  analysis  presents  a  summary  of  the  results  in  a 
single  graph  that  combines  measures  of  effectiveness  (MOEs)  for 
cost,  effectiveness,  risk  and  schedule.  The  MOEs  are  normalized, 
which  means  that  a  larger  number  indicates  a  better  system.  This 
makes  the  MOEs  for  cost,  risk,  and  schedule  inversely  related  to 
the  raw  measures  of  these  factors.  As  is  the  case  with  component 
analyses,  the  spreadsheets  may  be  used  to  get  a  more  precise  view 
of  the  numbers  that  are  shown  in  the  graph,  to  modify  any  of  the 
individual  input  data,  and  to  set  adjustment  weights. 

Document  requirements  and  analysis  results.  The  output 
documentation  provides  a  trace  or  audit  trail  of  selections  and 
analysis  results.  The  final  results  are  organized  in  outline 
form  which  may  be  printed  or  saved  as  a  text  file.  That  text 
file  can  then  be  edited  using  word  processing  software. 

The  outline  is  based  on  sections  required  for  the  TOD 
documentation.  The  user  may  edit  the  sections  included  in  the 
outline,  and  add  more  sections.  The  user  also  may  link  each 
analysis  to  one  or  more  sections.  For  example,  the  results  of 
the  function  selection  module  may  be  linked  to  the  "Requirements" 
section  of  the  outline.  When  the  outline  is  printed  out,  it  will 
contain  the  specific  functions  that  were  selected,  and  the  time 
that  they  were  selected. 

Each  analysis  in  the  CFP-Aid  contributes  to  the  outline 
being  generated.  Because  modules  can  be  used  several  times,  it 
is  possible  to  save  the  results  each  time,  producing  a  cumulative 
record  of  all  analyses  performed  using  the  CFP-Aid.  In  this  way 
the  outline  can  provide  an  audit  trail  of  the  analyses  performed 
for  the  project.  When  the  analyses  are  completed,  the  user  may 
regenerate  the  final  outline,  so  that  it  shows  only  the  most 
recent  results . 

CFP-Aid  Data 

Many  of  the  data  structures  for  the  CFP-Aid  come  directly 
from  the  OSBATS  model.  However,  those  data  structures  and  the 
data  manipulation  mechanisms  are  different  in  the  CFP-Aid. 
Specifically,  training  device  components  in  the  OSBATS  model  are 
selected  based  on  their  technical  performance  (interpreted  as 
learning  effectiveness)  compared  to  fidelity  or  instructional 
feature  requirements.  Specific  learning  and  transfer  models  were 
developed  to  assess  the  effectiveness  of  instructional  features 
and  fidelity  levels.  These  models  were  feasible  because  the  two 
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links  into  components  considered  by  the  OS BATS  model  were 
directly  related  to  learning  and  transfer  considerations.  These 
links  were  all  model  based  and  were  not  stored  in  the  data  base 
for  future  reference  or  use. 

The  CFP-Aid  data  has  four  links  to  components,  including 
links  from  tasks  and  cues,  in  addition  to  the  links  considered  by 
OSBATS.  Because  of  this  variety  of  uxnks  to  components,  it  was 
not  feasible  to  implement  the  learning  and  transfer  model 
evaluation  that  was  used  in  OSBATS.  Extension  of  that  model  to 
the  CFP-Aid  would  be  a  reasonable  extension  of  the  OSBATS  model. 

The  initial  design  for  the  CFP-Aid  incorporated  most  of  the 
data  variables  reguired  by  OSBATS.  However,  some  of  these 
variables  are  not  required  by  the  prototype  aid.  The  variables 
are  still  maintained  in  the  structure,  and  are  available  for 
future  enhancements  to  the  system. 

Evaluation 

The  evaluation  of  the  CFP-Aid  system  was  based  on  extensive 
interaction  with  engineers  during  system  use.  The  goal  of  the 
evaluation  was  to  have  users  evaluate  the  user  interface,  as  well 
as  the  functionality  and  usability  of  CFP-Aid.  We  also  evaluated 
the  system  by  obtaining  subjective  estimates  of  how  well  it 
supports  required  functions  in  the  TOD  process.  The  evaluation 
results  provide  the  basis  for  recommendations  to  management  for 
revision  and  use  of  the  aid  on  the  job. 

Approach 

Subjects.  Four  project  directors  from  PM-TRADE  participated 
in  the  study.  They  were  two  males  and  two  females  familiar  with 
training  device  analysis  and  design,  between  25  and  35  years  old, 
and  had  a  high  level  of  experience  and  comfort  working  with 
computers . 

Study  materials.  The  evaluation  required  the  subjects  to 
complete  a  series  of  surveys  and  questionnaires.  The  first 
survey  is  the  Interface  Evaluation  Checklist  (IEC) ,  by  Ravden  and 
Johnson  (1989) ,  which  evaluates  the  user  interface  for  usability 
along  nine  dimensions.  The  nine  dimensions  are  Visual  Clarity, 
Consistency,  Compatibility,  Informative  Feedback,  Explicitness, 
Appropriate  Functionality,  Flexibility  and  Control,  Error 
Prevention  and  Correction,  and  User  Guidance  and  Support.  The 
second  instrument  is  the  Acceptance  of  CFP-Aid  Survey,  a  modified 
version  of  a  set  of  questionnaires  created  by  Companion  (1990) . 
The  acceptance  survey  gathered  subjective  ratings  of  projected 
utility,  ease  of  use,  relevance  to  the  job,  and  effectiveness  in 
aiding  the  TOD  process.  The  last  instrument,  a  Process 
Questionnaire  developed  specifically  for  this  effort,  gathered 
information  about  the  level  of  task  and  subtask  support  provided 
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by  CFP-Aid  for  the  TOD  process.  The  questions  targeted  areas 
where  CFP-Aid  could  affect  time/effort,  influence  product 
quality,  change  data  requirements,  and  address  regular  processes 
or  activities. 

Procedures 

The  evaluators  participated  in  six  2-hour  sessions  conducted 
over  a  two-week  period.  The  evaluation  divided  the  six  sessions 
into  three  phases:  Training  Sessions,  Operation  Sessions,  and  a 
Termination  Session. 

Phase  I:  Training.  Phase  I  of  the  evaluation  provided  the 
evaluators  with  two  Training  sessions  that  used  the  CFP-Aid 
User's  Guide  (Elder,  Sticha,  Page,  and  Singer,  in  publication)  as 
training  material.  In  Training  Session  1,  the  evaluators 
followed  a  guided  walk  through  of  CFP-Aid' s  screens  and 
functions.  Training  Session  2  provided  the  evaluators  with  in- 
depth,  hands-on  experience  in  operating  CFP-Aid.  The  session 
required  evaluators  to  create  a  hypothetical  TOD  situation  and 
run  the  Requirements,  Components  and  Analysis  modules  of  CFP-Aid. 


Phase  II:  Operation.  Phase  II  of  the  evaluation  provided 
the  evaluators  with  three  operational  sessions  in  which  they 
recreated  a  TOD  situation  based  on  their  previous  experience. 

The  use  of  a  realistic  TOD  situation  provided  an  ecologically 
valid  context  to  the  evaluation.  By  the  end  of  the  Operational 
Phase  the  evaluators  had  viewed  each  screen  and  used  each  CFP-Aid 
function  a  minimum  of  three  times. 

In  Operation  Session  1  the  evaluators  began  encoding  the 
needed  database  items  (MOS,  ASI,  SQI,  tasks,  functions, 
components,  and  cues)  for  the  selected  TOD  project.  The  session 
required  the  evaluator  to  make  all  necessary  system  inputs,  the 
experimenter  provided  only  limited  guidance.  In  Operation 
Session  2 ,  the  evaluators  continued  work  on  the  selected  TOD 
project  by  performing  a  series  of  predefined  exercises  designed 
to  explore  the  Requirements,  Component,  and  Documentation  modules 
of  CFP-Aid.  The  second  session  ended  with  the  evaluators 
completing  three  sections  of  the  IEC.  In  Operation  Session  3  the 
evaluators  completed  work  on  the  TOD  project  by  performing 
additional  exercises  within  the  Analysis  and  Documentation 
modules  of  CFP-Aid.  The  third  session  ended  with  the  evaluators 
examining  the  output  document  produced  by  CFP-Aid,  and  completing 
five  sections  of  the  IEC. 

Phase  III:  Termination.  The  Termination  Session  did  not 
require  the  evaluator  to  operate  CFP-Aid.  Rather,  the  evaluators 
completed  the  remaining  sections  of  the  IEC,  and  filled  out  the 
Process  Questionnaire  and  the  Acceptance  Survey.  A  unstructured 
interview  was  then  conducted  to  obtain  the  evaluator's  general 
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opinions  on  the  usability  and  acceptance  of  CFP-Aid  in  their 
organization. 

Results 


The  number  of  respondents  (four)  used  in  the  evaluation 
precludes  the  use  of  standard  statistical  analyses.  Therefore, 
the  results  section  presents  a  summary  of  the  evaluator's  median 
or  modal  responses  to  the  Interface  Evaluation  Checklist,  the 
Acceptance  Survey,  and  the  Process  Survey.  As  an  example,  if  one 
rating  was  very  satisfactory,  two  were  moderately  satisfactory, 
and  one  was  neutral;  moderately  satisfactory  is  reported  for  the 
dimension.  A  review  of  the  evaluator's  overall  consensus  (the 
most  frequent  comment)  about  the  CFP-Aid,  obtained  in  the 
Termination  Session,  is  provided  last. 

Interface  Evaluation  Checklist.  The  Interface  Evaluation 
Checklist  (Ravden  and  Johnson,  1989)  provided  ratings  about  nine 
different  interface  characteristics.  The  evaluator's  rating  of 
the  characteristics  are  presented  below,  with  an  identification 
of  System  Usability  Problems. 

The  evaluators  rated  the  overall  level  of  system 
Consistency,  Flexibility  and  Control ,  Explicitness,  and  User 
Guidance  and  Support  as  moderately  satisfactory.  The  application 
of  color  codes,  screen  formats,  control  actions,  and  cursor 
locations  remained  consistent  throughout  the  system.  The 
evaluators  found  it  easy  to  "undo”  actions,  and  to  step  back  to  a 
previous  processing  stage.  The  users  believed  that  the  system 
provided  an  acceptable  amount  of  control  when  requesting 
information,  and  when  carrying  out  a  sequence  of  activities.  The 
menu  structure  was  judged  to  be  an  easy  means  of  navigation. 

Although  the  evaluators  rated  these  characteristics  as 
moderately  satisfactory,  they  noted  some  deficits.  The  first  was 
that  users  could  not  tailor  the  interface  system,  color  codes 
were  not  flexible,  and  the  system  did  not  provide  shortcuts  for 
experienced  users.  These  are  relatively  harch  criteria  for  a 
developmental  prototype  system.  A  more  relevant  comment  provided 
by  the  evaluators  was  that  the  system  provided  inadequate  on-line 
help  facilities.  In  addition,  the  hard  copy  user  guide  was 
judged  to  have  insufficient  in  depth  coverage,  and  did  not  always 
provide  adequate  explanations  concerning  user  and  system  errors. 
As  a  result,  the  evaluators  occasionally  had  difficulty 
understanding  the  jargon  used  by  CFP-Aid. 

The  evaluators  rated  the  level  of  system  Functionality  as 
moderately  satisfactory  to  neutral.  The  evaluators  judged  that 
the  CFP-Aid  possessed  appropriate  screen  formats  and  the 
functionality  required  to  support  system  tasks.  The  sequence  of 
activities  required  to  complete  a  task  paralleled  user 
expectations  and  perceptions  of  the  tasks.  The  neutral  portion 
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of  the  rating  stemmed  from  the  evaluators  being  unsure  of  what 
stage  CFP-Aid  had  reached  when  processing  a  command.  The  system 
did  not  provide  all  the  appropriate  task  information  at  start  up, 
nor  did  it  allow  them  to  access  this  information  when  needed. 
Lastly,  the  reasoning  for  sequencing  certain  screens  was  not 
transparent,  resulting  in  a  lack  of  immediate  comprehension  for 
the  overall  structure  of  CFP-Aid. 

The  evaluators  rated  the  level  of  Informative  Feedback,  and 
Error  Prevention  and  Correction  provided  by  the  system  as 
neutral.  This  rating  was  due  to  instructions  and  CFP-Aid  prompts 
that  were  unclear,  and  error  messages  that  did  not  clearly 
explain  why  or  where  an  error  occurred.  When  the  system 
identified  an  error  the  steps  needed  to  correct  the  error  were 
unclear.  Status  messages  about  system  processing  were  perceived 
to  be  static  and  uninformative.  The  messages  did  not  provide 
time  to  completion  information  nor  clearly  inform  the  evaluators 
of  the  completion  of  a  requested  action.  The  users  felt  that  the 
system  insufficiently  validated  inputs  and  did  not  detect  or 
correct  input  errors  before  processing.  The  system  provided  few 
error  blocks,  and  also  permitted  unauthorized  actions  by  the 
evaluators.  Errors  made  in  one  section  of  the  system  could  cause 
undetected  processing  errors  in  other  modules.  However,  the 
system  was  judged  adequate  in  protecting  against  the  most  common 
errors . 

The  evaluators  rated  the  Visual  Clarity  of  the  prototype  as 
very  satisfactory,  and  the  level  of  system  Consistency  as  very  to 
moderately  satisfactory.  Appropriate  screen  formats,  and 
necessary  functionality,  provided  adequate  support  to  complete 
tasks.  The  use  of  menu  panel  titles  presented  clearly 
identifiable  screens,  and  the  logical  organization  of  the  screens 
into  clearly  aligned  columns  made  the  presented  information  easy 
to  see  and  read.  The  format  of  the  displayed  information 
followed  established  conventions  (dates,  telephone  numbers, 
etc...),  and  used  units  that  the  user  normally  worked  with 
(dollars,  meters,  scales) .  Cursor  movement  corresponded 
directly  to  user  inputs,  and  were  similar  to  those  encountered  in 
other  systems.  In  general  the  evaluators  felt  that  CFP-Aid 
provided  uncluttered  screens,  used  color  effectively,  and 
presented  cleanly  drawn  graphics. 

When  asked  about  System  Usability  Problems  the  evaluators 
judged  that  the  system  functioned  satisfactorily,  although  some 
minor  problems  were  identified  (see  above) .  The  limited  set  of 
screen  colors  were  judged  as  making  the  system  screens  easy  to 
watch,  system  response  times  were  judged  appropriate,  information 
stayed  on  screen  long  enough  read,  and  the  evaluators  had  no 
difficulty  understanding  what  was  going  on.  The  input  devices 
were  judged  easy  to  use,  and  the  screen  formats  presented  most  of 
the  required  task  information.  These  factors  were  judged  as 
keeping  the  user's  memory  requirements  low. 
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User  Acceptance  Survey.  The  Acceptance  Survey  for  the  CFP- 
Aid  provided  ratings  for  seven  different  areas;  Overall  Reaction, 
User  Acceptance,  Screens,  Terminology,  Learning  to  use  CFP-Aid, 
Using  CFP-Aid,  and  reactions  to  the  CFP-Aid  Output.  Again,  the 
results  are  reported  in  terms  of  the  evaluators  consensus. 
Consensus  was  judged  as  before  with  the  most  freguent  or  central 
response  being  reported. 

The  evaluator's  Overall  Reaction  to  the  usefulness  and 
productivity  of  the  system  was  neutral.  Their  ratings  indicated 
that  they  found  the  system  easy  to  use  and  useful,  but  thought 
that  the  system  provided  inadequate  power  and  flexibility.  (This 
comment  seems  to  reflect  their  comprehension  of  the  system  as 
fielded  software  rather  than  protype  software.)  In  accordance 
with  those  ratings  was  the  rating  of  CFP-Aid 's  level  of  potential 
User  Acceptance.  In  general  they  indicated  comfort  in  working 
with  CFP-Aid,  judging  that  it  might  increase  job  effectiveness. 
Although  they  could  work  with  CFP-Aid,  they  also  indicated  that 
they  could  conduct  a  TOD  as  well  using  current  STRICOM 
procedures . 

The  evaluator's  reaction  to  the  CFP-Aid  Screens , 

Terminology ,  and  Learning  to  Use  CFP-Aid  were  positive.  The 
evaluators  found  the  menu  screens  to  be  logical  organized  and 
clearly  presented.  The  ratings  showed  that  the  labels  used  for 
the  different  functions  were  clear,  and  the  information  prompts 
were  moderately  helpful.  The  terminology  used  was  judged 
consistent,  but  not  judged  to  be  very  helpful.  The  evaluators 
found  it  easy  to  explore  and  learn  new  features  within  the 
system.  However,  they  did  feel  CFP-Aid  was  lacking  in  the  amount 
of  instructional  material  provided. 

The  evaluators  ratings  indicated  that  Using  CFP-Aid  was  not 
too  difficult,  and  they  felt  that  the  system  performed  tasks  in  a 
straight  forward  manner.  The  evaluators  judged  the  feedback 
provided  as  acceptable.  The  memory  requirements  for  the 
evaluators  was  judged  to  be  low,  and  the  error  messages  were 
considered  to  be  helpful,  when  provided.  Overall,  the  reactions 
of  the  evaluators  to  the  CFP-Aid  outputs  were  negative.  They 
claimed  that  the  presentation  format  of  the  outputs  made  it 
difficult  for  them  to  determine  the  usefulness  of  the  outputs. 

The  format  was  judged  to  be  confusing  and  difficult  to  understand 
by  two  of  the  four  evaluators. 

Process  Survey.  The  Process  Survey  provided  information 
about  the  analyst's  reactions  to  CFP-Aid  in  three  distinct  phases 
of  the  TOD  process:  Data  Collection,  TOD  Analysis,  and 
Documentation.  A  synopsis  is  provided  for  each  topic  area. 

The  evaluator's  ratings  indicated  that  CFP-Aid  had  the 
potential  to  aid  the  collection  of  information  for  the  TOD.  They 
believed  that  use  would  increase  slightly  the  number  of  hours 
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required  to  do  a  TOD,  in  order  to  accumulate  the  additional  data. 
However,  they  also  judged  that  using  the  system  might  improve  a 
user's  technical  understanding  of  a  new  device  and  related 
systems.  The  CFP-Aid  was  judged  to  have  the  potential  to  help 
obtain  both  user  and  school  inputs  when  determining  training 
device  requirements  and  identifying  task  and  functions  to  train. 
The  evaluators  felt  that  CFP-Aid  showed  a  potential  to  identify 
existing  data  sources,  and  organize  and  retain  this  data  once 
gathered.  The  evaluators  judged  that  this  would  aid  in  the 
identification  of  technical  options  while  conducting  the  market 
survey.  Overall,  the  projected  increase  in  time  to  perform  seems 
to  be  balanced  by  the  increase  in  detail,  improved  understanding 
of  requirements,  and  the  aid  in  obtaining  inputs  for  the  training 
requirements . 

Ratings  indicated  that  the  evaluators  felt  that  CFP-Aid 
could  provide  assistance  when  conducting  a  TOD  analysis.  They 
judged  that  the  system  might  assist  in  identifying  and  organizing 
existing  data  and  previously  gained  knowledge.  They  indicated 
that  conducting  a  technical  risk  analysis  using  the  CFP-Aid  may 
improve  the  identification  of  new  and  existing  technologies  as 
system  options.  Their  ratings  showed  that  they  felt  the  system 
provided  assistance  when  conducting  technical  risk  reviews  and 
prototype  tests.  Although  the  system  doesn't  provide  assistance 
in  determining  contractor  quality  or  in  the  determination  of 
component  availability  (which  were  not  design  goals) ,  the  system 
was  judged  to  provide  some  assistance  in  conducting  schedule  vs. 
requirement  analysis,  and  cost  vs.  requirement  analysis. 

The  evaluators  provided  a  variety  of  responses  concerning 
the  output  document  produced  by  CFP-Aid.  They  gave  positive 
ratings  concerning  the  potential  of  the  output  to  assist  in 
creating  the  TOD  documentation,  tracking  required  trade-offs,  and 
in  providing  additional  detail  to  the  documentation.  They  felt 
that  the  CFP-Aid  could  provide  the  strongest  assistance  and  value 
by  contributing  as  a  technical  resource.  Otherwise  the 
evaluators  were  neutral  concerning  CFP-Aid 's  potential  ability  to 
assist  in  the  identifying  security  needs,  Tempest  requirements, 
or  the  consideration  of  other  agencies  inputs  (items  present  on 
the  surveys  and  questionnaires  that  were  irrelevant  to  the  design 
goals,  and  recognized  as  such) . 

Conclusion 

Although  the  CFP-Aid  incorporates  many  of  the  analyses  and 
data  from  the  OSBATS,  there  are  several  critical  distinctions 
between  the  two  models.  The  primary  distinction  is  that  the  CFP- 
Aid  design  is  based  on  existing  analysis  requirements,  and 
addresses  the  expressed  needs  of  individuals  that  are  responsible 
for  the  required  analyses.  This  feature  makes  the  CFP-Aid 
different  from  any  existing  cost  effectiveness  model  for  training 
device  design  or  evaluation. 
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The  second  critical  distinction  is  that  the  CFP-Aid  supports 
multiple  methods  of  inferring  the  need  for  specific  system 
components.  Other  models,  such  as  OSBATS,  support  a  single  line 
of  reasoning  from  task  characteristics  to  device  requirements. 

The  multiple  links  in  the  CFP-Aid  and  the  possibility  for 
multiple  rule  bases  allow  for  a  very  rich  analysis  structure  that 
can  use  the  data  that  are  available  as  the  basis  of  its 
recommendations . 

Discussion  of  Evaluation  Results 

The  goal  of  the  evaluation  was  to  have  prospective  users 
evaluate  the  user  interface,  functionality,  and  usability  of  the 
CFP-Aid.  Overall,  the  results  indicated  positive  to  neutral 
ratings  for  all  major  categories,  except  for  the  unsatisfactory 
rating  for  level  of  informative  feedback  provided.  The  results 
show  that  the  system  has  an  adequate  user  interface,  adequate 
coherent  functioning,  and  potential  in  assisting  the  user  during 
the  TOD  process.  The  results  also  provide  important  information 
for  guiding  future  development. 

The  user  interface  supports  the  encoding  of  collected  data, 
and  aids  the  user  in  conducting  the  required  analyses.  The 
interface  contains  few  surprises  and  provides  adequate  cues  for 
the  users.  The  method  of  input  is  consistent,  and  the 
information  presented  in  the  screens  is  formatted  clearly.  The 
appropriate  use  of  color,  text  columns,  labels,  and  user 
expectations  in  the  design  of  the  interface  produced  a  software 
package  that  was  easy  to  use. 

In  general  the  system  functions  according  to  user 
expectations  and  provides  adequate  support  for  TOD  procedures. 
Although  the  prototype  functioned  well,  several  system  deficits 
were  identified.  The  biggest  hindrance  to  system  operation  is 
the  lack  of  helpful  support  documentation,  both  on  and  off-line. 
Unfortunately,  GURU  (Micro  Data  Base  Systems,  Inc.,  1991)  does 
not  support  context  dependent  on-line  help  features.  The  off¬ 
line  documentation  is  being  revised,  and  can  easily  be  improved. 
The  outline  documentation  produced  by  the  prototype  seems  to  lead 
to  user  confusion,  although  the  output  matches  "good"  TOD's.  The 
rigid  output  format,  and  the  large  amount  of  information 
included,  seemed  to  make  the  documentation  difficult  to  read  and 
understand.  All  of  these  issues  can  be  rectified  through  user 
guidance  and  intervention. 

It  is  worth  noting  that  the  evaluators  the  felt  CFP-Aid 
could  be  used  to  train  a  new  engineer  on  the  TOD  process.  A 
tutorial  using  the  CFP-Aid  could  provide  work  exercises  using  an 
existing  TOD  to  provide  the  new  engineer  with  more  structure  and 
guidance  than  is  presently  provided.  Use  of  the  prototype  in 
this  way  may  reduce  the  time  required  by  senior  staff  to  guide 
the  new  engineers  through  the  TOD  process.  This  in  itself  may 
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provide  enough  benefit  to  support  the  further  refinement  or 
development  of  CFP-Aid. 

The  Challenge  of  Implementation 

Because  it  was  designed  for  an  existing  part  of  the  training 
device  design  process,  the  CFP-Aid  should  avoid  many  of  the 
barriers  to  adoption  that  were  experienced  with  OSBATS  and  other 
training  cost  effectiveness  models.  However,  there  are  still 
substantial  challenges  to  the  use  of  the  CFP-Aid  as  a  regular 
part  of  the  Concept  Formulation  Process.  These  challenges 
concern  a  user's  understanding  of  the  CFP-Aid  system,  the 
requirements  for  data  entry,  and  the  need  for  output  tailored  to 
the  desires  of  the  user.  A  major  challenge  also  arises  from  the 
non- standardized  requirements  of  the  Concept  Formulation  Process 
as  practiced  at  STRICOM. 

Even  though  the  design  of  the  CFP-Aid  was  based  on 
engineer's  descriptions  of  the  TOD  process,  it  contains  several 
novel  concepts  that  were  not  taken  directly  from  current 
practices.  For  example,  the  CFP-Aid  has  a  much  greater  emphasis 
on  tasks  and  functions  than  was  mandated  from  user  interviews  and 
TOD  documentation.  Furthermore,  the  multiple  paths  between 
modeling  elements  is  slightly  different  from  current  procedures, 
as  well  as  from  any  other  model  of  which  we  are  aware.  As 
mentioned  above,  it  will  be  necessary  to  ensure  that  users  have  a 
firm  understanding  of  the  processes  supported  by  the  CFP-Aid  and 
the  methods  which  support  those  processes. 

The  CFP-Aid  requires  significant  data  entry  when  it  is  used 
for  the  first  few  times  in  a  domain.  As  it  is  used  and  the 
databases  grow,  the  requirements  for  new  data  should  decrease. 
However,  initial  data  entry  may  be  a  barrier  to  adoption  of  the 
aid.  Consequently,  we  recommend  that  ARI  assist  the  early 
applications  of  the  aid  by  providing  data  entry  support. 

Closely  related  to  the  data  entry  issue  is  the  issue  of 
maintaining  and  assuring  the  integrity  of  the  databases.  The 
prototype  software  is  designed  for  individual  use,  thus 
minimizing  the  concern  for  database  maintenance.  However,  an 
operational  CFP-Aid  would  be  used  by  many  people  who  would  share 
databases.  Consequently,  procedures  for  data  maintenance, 
specification  of  system  administrator  roles,  and  database  update 
procedures  will  need  to  be  developed. 

The  CFP-Aid  provides  a  uniform  procedure  for  conducting  the 
TOD.  Although  there  is  considerable  flexibility  in  the  operation 
of  the  aid,  it  is  unlikely  that  the  aid  will  accommodate  the 
complete  range  of  TOD  analyses  that  are  currently  conducted  by 
STRICOM.  Limits  in  the  flexibility  will  need  to  be  identified 
and  reduced  or  eliminated  in  any  operational  version  of  the  CFP- 
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Aid.  In  addition,  it  will  be  beneficial  to  establish  a  greater 
degree  of  uniformity  in  the  TOD  process. 

Future  Research  and  Development 

Future  research  and  development  of  the  CFP-Aid  will  depend 
on  the  acceptance  of  the  prototype  by  STRICOM.  It  is  likely  that 
use  of  the  prototype  software  will  lead  to  the  generation  of  a 
list  of  research  and  development  needs  that  is  much  more  useful 
than  any  list  that  could  be  developed  at  this  time. 

Nevertheless,  there  are  several  areas  for  potential  further 
development  that  can  be  mentioned  as  a  result  of  the  development 
efforts. 

•  Increased  flexibility  in  analyses  and  output.  This  option 
would  allow  the  user  greater  flexibility  in  performing  CERS 
analysis,  including  the  ability  to  specify  the  factors  that 
are  considered  and  to  format  analysis  output  according  to 
user  specifications.  Current  capabilities  allow  the  user  to 
modify  the  weights  of  a  fixed  set  of  factors  within  a  single 
analysis  format. 

•  More  comprehensive  effectiveness  model.  This  option  would 
combine  information  from  tasks,  cues,  instructional 
features,  and  fidelity  issues  to  obtain  a  more  precise 
measure  of  effectiveness  than  is  currently  used. 

•  Families  of  tasks  and  cues.  In  the  prototype  CFP-Aid, 
components  are  organized  into  families  that  perform  similar 
functions,  such  as  visual  display,  image  generation,  and  so 
forth.  This  option  would  extend  that  organization  to  tasks 
and  cues  in  order  to  provide  the  user  with  greater 
flexibility  in  reviewing  and  selecting  them. 

•  Flexible  rule  base  operation.  Increasing  the  flexibility  of 
the  rule  base  design  should  make  the  rules  applicable  to  a 
wider  variety  of  functions,  allow  the  user  more  capability 
to  adjust  rule  base  output,  and  allow  the  application  of  the 
rule  bases  to  be  tailored  to  the  specific  needs  of  the  TOD 
being  performed. 

•  Addition  of  rule  bases.  Adding  rule  bases  to  the  system  is 
fundamental  to  its  ability  to  adequately  address  real  world 
training  domains.  Applications  could  be  pursued  using  the 
linking  functions  in  the  database,  but  this  would 
drastically  reduce  the  benefits  to  be  accrued  from  using  the 
system. 

Final  Comments 


Any  second  generation  system  should  improve  upon  the  user 
documentation,  provide  a  complete  database  of  tasks,  cues,  and 


components  with  the  system,  and  provide  more  thorough  guidance  in 
conducting  an  analysis.  These  simple  changes  will  increase  the 
level  of  user  confidence  and  result  in  better  acceptance.  Other 
than  improving  the  documentation  the  following  capabilities  could 
be  provided:  a)  a  source  list  for  cost,  risk,  effectiveness,  and 
schedule  data,  b)  allow  users  to  verify  current  database 
information,  c)  provide  a  means,  or  set  up  a  procedure  for 
automatic  database  updates,  d)  provide  system  short-cuts  for 
experienced  users,  and  e)  improve  the  error  protection  and 
correction  procedures. 

Although  the  evaluators  judged  CFP-Aid  capable  of  assisting 
design  engineers  in  conducting  a  TOD,  they  recognized  that  its 
use  will  change  their  job.  The  prototype  provides  a  stable 
structure  and  requires  specific  types  of  information  through 
implementing  standard  procedures.  Presently,  to  conduct  a  good 
TOD  requires  tedious  record  keeping  and  vast  amounts  of 
experience  within  a  structure  that  is  far  from  standard.  The 
CFP-Aid  provides  help  identifying  and  collecting  needed 
information,  and  helps  determine  priorities  for  its  use.  With 
CFP-Aid  the  information  requirements  are  fixed  and  sources  for 
the  information  are  identified.  If  the  system  is  implemented, 
what  used  to  be  a  highly  variable  process  will  become  a  short  set 
of  easily  accomplished  tasks. 
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